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DETAILED ACTION 

Response to Amendment 

1 . Applicants amendment filed February 8th 2006 has been entered and 
made of record. 

2. Applicant has amended independent claims 1 , 12 and 18. Claims 1-29 
are pending. 

3. Applicant's remarks in view of the submitted amendment are considered 
persuasive. However, the amendments presented now change the scope of the claims 
and require further consideration and a new rejection. 

4. Applicant has amended the independent claims 1,12 and 18 to include 
the limitation of wherein the three or more images are captured simultaneously. 

The reference of U.S. Patent 5,852,672 to Lu has been cited below to teach this 
limitation It should be noted that the determining 3-0 depth of a plurality of pixels in 
the base image by matching correspondence to a plurality of pixels in three or 
more images, each image representing one of the at least three views of the 
scene as claimed is performed on image data and the limitation of whether or not the 
images were captured at the same moment or not does not affect the function of 
determining depth information from multiple images. 

A new rejection has accordingly been made as necessitated by the amendment. 
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Previously Indicated Allowable Subject Matter 

Claim 29 was previously indicated allowable, however after further consideration, 
Claim 29 is now rejected under 35 USC 101 as non-statutory patentable subject matter. 

Claim Rejections - 35 USC § 101 

Claim 29 is rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter. Claim 29 is drawn to a computer-implemented 
process that merely manipulates data or an abstract idea, or merely solves 
mathematical problem without a limitation to a practical application in the technological 
arts. Namely, claim 29 discusses recovering depth information in the preamble of the 
claim as the main purpose of the method, but does specifically connect depth 
information to a tangible or concrete result. Claim 29 appears to only disclose a 
manipulation or calculation of data with no intent of purpose or way of using the result of 
such calculation. 

In order for a claimed invention to accomplish a practical application, it must 
produce a "useful, concrete and tangible result" State Street, 148 F.3d at 1373, 47 
USPQ2d at 1601-02 (see MPEP 2106.II.A). a practical application can be achieved 
through recitation of "a physical transformation outside the computer for which a 
practical application in the technological arts is either disclosed in the specification 
within the technological arts is either disclosed in the specification or would have been 
known to a skilled artisan", or "limited to a practical application within the technological 
arts" (MPEP 2106 IVB2(b)). Currently, claim meets neither of these criteria. In order 
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for the claimed process to produce a "useful, concrete and tangible" result, recitation of 
one or more of the following elements is suggested: 

• The manipulation of data that represents a physical object or activity 
transformed from outside the computer (MPEP 2106 l\/B2(b)(i)). 

• A recitation of a physical transformations outside the computer, for 
example in the form of pre or post computer processing activity (MPEP 
2106 IVB2(b)(i). 

• A direct recitation of a practical application in the technological arts (MPEP 
2106 IVB2(b)(ii)). 

Appropriate correction is required. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1, 4-9, 12-18 and 21-28 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over the combination of the publication by [Chen, VSMI\/I99] (Chen and 
Medioni , "A Volumetric Stereo Matching Method: Application to Image-Based 
Modeling", IEEE 1999) and U.S. Patent 5,853,672 to Lu. 
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With regard to claim 1 , [Chen, \/SMM99] describes a volumetric stereo matching 
method. The method involves the recovery of depth information - namely, the disparity 
surface d(u,v) (see [Chen, VSMM99]: Abstract, sentence 1; page 29, right column, last 
paragraph^ sentences 1-3; and page 30, left column, paragraph 1, sentences 2-3) - for 
pixels of a base image representing a view of a scene. The method comprises at least 
the following steps: 

(1 .a.) Detecting a plurality of pixels in a base image that represents a first 
view of a scene wherein the first view is one of at least three views 
of a scene (e.g. capturing images from various viewpoints of the 
scene). It should be apparent from the various stereo images 
shown in [Chen, VSMM99] that the volumetric stereo matching 
method involves this step. Any one of these images can be 
arbitrarily denoted as a "base image". See Also Example 4 on 
page 34, where 6 views of a teapot are used. 
(1 .b.) Determining 3-D depth of the plurality of pixels in the base image 
by matching correspondence to a plurality of pixels in three or more 
images each image representing one of the at least three views of 
the scene wherein each of the at least three views of the scene are 
situated in a non-linear arrangement and are further oriented in a 
plurality of non-parallel planes relative to each other. See Also 



1 



When referring to paragraphs in the cited references, the convention followed here is that the paragraph number is assigned to 
paragraphs of a given column (if applicable) or section, sequentially, beginning with the first full paragraph. Paragraphs that 
carry over to other columns will be referred to as the last paragraph of the column in which they began. 



Application/Control Number: 09/927,558 Page 6 

Art Unit: 2624 

Example 4 on page 34, where 6 views of a teapot are used. Clien 
discloses using views of a teapot as it is rotated, the views of which 
would therefore not lie on parallel planes. 
Steps (1 .a.)-(1.b) are typical of any stereo matching applications. The "matching 
correspondence" is discussed in Section 3.1 of [Chen, VSMM99]. The cross-correlation 
nu,v,d\s indicative of the matching correspondence between pixels of different images, 
each of which are representative of disparate views of the scene. 

(1 .c.) Tracing pixels (i.e. surface tracing) in a virtual piecewise continuous 
depth surface (i.e. disparity surface d{u,v)) by spatial propagation 
([Chen, VSMM99] section 3.2, paragraph 1, sentence 4) starting 
from the detected pixels in the base image (i.e. all pixels (a^v) of the 
base image) by using the matching and corresponding plurality of 
pixels in the three or more images (e.g. seed pixels that satisfy a 
given cross-correlation criterion - [Chen, VSMM99] page 31, right 
column, paragraph 1, sentences 1-2) to create the virtual piecewise 
continuous depth surface viewed from the base image. Please refer 
to Section 3 of [Chen, VSMM99]. 
(1 .d.) Each successfully traced pixel is associated with a depth in the 

scene viewed from the base image. This follows from the fact that 
traced pixels lie on the disparity surface. Again, the disparity 
surface is indicative of the depth of all visible surfaces of a given 
scene (page 29, right column, last paragraph, sentences 1-3). 
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In this manner, the a volumetric stereo matching method of [Chen, VSMM99] conforms 
sufficiently to the method proposed by the Applicant in claim 1 . 

Applicant has amended the independent claims 1,12 and 18 to include the 
limitation of wherein the three or more images are captured simultaneously. 

[Chen, VSMM99] does not explicitly disclose wherein the three images are 
captured simultaneously. 

Lu teaches the simultaneous capture of multiple images (column 3, lines 56-64 
and column 6, lines 1-10 and Fig. ) for use in an embodiment similar to that of [Chen 
\/SMM99]. In the invention of Lu, the advantage of capturing the images at a single 
moment in time is advantageous in order to ensure that objects being captured do not 
move or shift between image capture from different images. Therefore it would have 
been obvious to one of ordinary skill in the art at the time of invention to capture multiple 
images simultaneously for generating 3D image data from matching the different 
images in order to ensure that the images represent the objects in the same position so 
that all of the images to be matched represent the same set of data defining the object 
in combination with the 3D depth recovery information disclosed by [Chen VSMM99]. 

The following is in regard to Claim 18. Claim 18 recites essentially the same 
limitations as claim 1 . (The 3D reconstruction instructions merely implement the depth 
recovery method of Claim 1 ). Therefore, with regard to claim 1 8, remarks analogous to 
those presented above relative to claim 1 are applicable. 
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The following is in regard to Claim 4. As shown above, the combination of [Chen, 
VSMM99] and Lu discloses a volumetric stereo matching method that adequately 
conforms to the depth recovery method of Claim 1 . The volumetric stereo matching 
method of [Chen, VSMM99] further involves: 

Propagating a front of a virtual piece of a continuous depth surface 
(e.g. a surface front - this terminology is used in sentence 5 of the 
last paragraph of the right column on page 31) to at least one 
neighboring pixel (e.g. each 4-neighbor of {u,v) - Fig. 4 of [Chen, 
VSMM99]) starting from the detected pixels (e.g. pixel {u,v)) in the 
base image. 

It should be apparent from the algorithm listed in Fig. 4 of [Chen, VSMM99] that the 
boundary (front) of the disparity surface is propagated. In this manner, the volumetric 
stereo matching method of [Chen, VSMM99] conforms sufficiently to the method 
proposed by the Applicant in Claim 4. 

The following is in regard to Claim 21. Claim 21 recites essentially the same 
limitations as claim 4. (The 3D reconstruction instructions merely implement the depth 
recovery method of Claim 4). Therefore, with regard to claim 21 , remarks analogous to 
those presented above relative to claim 4 are applicable. 

The following is in regard to Claims 5-6. As shown above, the combination of 
[Chen, VSMM99] and Lu discloses a volumetric stereo matching method that 
adequately conforms to the depth recovery method of Claim 4. Tracing the disparity 
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surface according to the volumetric stereo matching method of [Chen, VSMM99] further 
involves the following: 

(5.a.) Determining when a boundary is reached between two propagating 
fronts of virtual pieces of a continuous depth surface ([Chen, 
VSMM99] page 31, right column, last paragraph, sentence 5). 
If the boundaries of two propagating fronts coincide then: 

(6. a.) Comparing the cross-correlations of the two propagating fronts 
about the reached boundary. This comparison is performed, for 
example, in the following step shown in [Chen, VSMM99] Fig. 4: 

disparity(i7>') = nu\v\d:oa uy.dyud' UdV 
Clearly, a comparison of the cross-correlations, pa'^v^cf.'andp 
u\v\d^[ is tantamount to a comparison of the matching costs of the 
two surface fronts 
(6.b.) The surface front with the greatest cross-correlation prevails 

([Chen, \/SMM99] page 31, right column, last paragraph, sentence 
5). In other words, the propagation of the front with the lower cross- 
correlation is stopped. 
Step (6.b.) is equivalent to stopping the propagation of the front with the higher 
compared matching cost^. In this manner, the a volumetric stereo matching method of 

^ To see this recall how the Applicant defines matching cost (page 9, lines 10-15 of the Applicant's disclosure and, in 
particular, equation (8) on page 17). The matching cost is low for pixels that are likely matches and is high for unlikely 
matches. The relationship between the matching cost and cross-correlation (NCC) is shown in the Applicant's equation (8). 
Clearly, a high NCC results in a low matching cost Eig) and indicates a likely match, whereas a low NCC results in a high 
matching cost and indicates an unlikely match. Therefore, by stopping the propagation of the front with the lower NCC, one 
has effectively stopped the propagation of the front with the higher matching cost. 
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[Chen, VSMM99] conforms sufficiently to both the methods of Claim 5 and the method 
of Claim 6. 

The following is in regard to Claim 22-23. Claims 22-23 recite essentially the 
same limitations as claim 5-6, respectively. (The 3D reconstruction instructions of these 
claims merely implement the depth recovery methods of Claim 5-6). Therefore, with 
regard to claim 22-23, remarks analogous to those presented above relative to claim 5- 
6 are respectively applicable. 

The following is in regard to Claim 7. As shown above, the combination of [Chen, 
VSMM99] and Lu discloses a volumetric stereo matching method that adequately 
conforms to the depth recovery method of Claim 1 . Tracing the disparity surface 
according to the volumetric stereo matching method of [Chen, VSMM99] further 
involves the following: 

Propagating a front of a virtual piece of a continuous depth surface 
to at least one neighboring pixel surrounded by a predefined size 
window in the continuous depth surface. According to the 
volumetric stereo matching method of [Chen, VSMM99], the 
normalized cross-correlation (NCC) over a window (e.g. one with 
dimensions and centered at pixels {u,v) and (iv+d,v)) is used as the 
similarity measurement. See Section 3.1 of [Chen, VSMM99]. 
In this manner, the a volumetric stereo matching method of [Chen, VSMI\/I99] conforms 
sufficiently to the method of Claim 7. 
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The following is in regard to Claim 24. Claim 24 recites essentially the same 
limitations as claim 7. (The 3D reconstruction instructions merely implement the depth 
recovery method of Claim 7). Therefore, with regard to claim 24, remarks analogous to 
those presented above relative to claim 7 are applicable. 

The following is in regard to Claim 8-9 and 25-26, As shown above, the 
combination of [Chen, VSMM99] and Lu discloses a volumetric stereo matching method 
that adequately satisfies the limitations of Claims 7 and 24. The limitations set forth in 
Claims 8-9 and 25-26 were addressed previously in the discussion above relating to 
Claims 5-6 and 22-23. The details will not be repeated here. 

The following is in regard to Claim 12. [Chen, VSMM99] discloses a volumetric 
stereo matching method that includes the following: 

(12. a.) Providing a plurality of seed pixels that represent 3-D depth of the 
plurality of pixels in the base image view of a scene (i.e. seed 
voxels - [Chen, VSMM99] page 31 , right column, paragraph 1 ). 
Seed voxels are determined by matching correspondence to a 
plurality of pixels in three or more images representing a plurality of 
views of the scene, wherein the base image is one of at least three 
views of the scene, each of the at least three views of the scene 
being situated in a non-linear arrangement and are further oriented 
in a plurality of non-parallel planes relative to each other (i.e. 
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ensuring that the cross-correlation Du,v,d between the views 
satisfies au,v,dnt1, t1 being a predetermined threshold - [Chen, 
VSMM99] page 31, right column, paragraph 1, lines 3-4). See also 
example 4 on page 34 with the 6 rotating views of the teapot. 
(12,b.) Tracing pixels in a virtual piecewise continuous depth surface by 
spatial propagation starting from the provided plurality of seed 
pixels in the base image by using the matching and corresponding 
plurality of pixels in the three or more images to create the virtual 
piecewise continuous depth surface viewed from the base image, 
each successfully traced pixel being associated with a depth in the 
scene viewed from the base image. The surface tracing was 
treated above with respect to step (1 .c). See also [Chen, VSMM99] 
right column, paragraph 2 and Fig. 4. 
[Chen, VSMM99] presents the volumetric stereo matching method entirely as an 
algorithm and forgoes any mention of possible hardware implementations. However, the 
volumetric stereo matching method is shown to have been executed on a computer 
(e.g. SGI workstation - [Chen, VSMM99] page 34, left column, paragraph 2). In this 
case, item (12. a.) would be implemented in computer-readable instructions, say as a 
"image matchjng module", and item (12. b.) would be implemented in computer-readable 
instructions, say as a "propagation module". The "image matching module" and 
"propagation module" would presumably be resident (electrically coupled) in some 
memory which is, in turn, coupled to a central processing unit (CPU). In this manner, a 
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computer implementation of the a volumetric stereo matching method of [Chen, 
VSMM99] would sufficiently conform to the image processing system of Claim 12, 

Applicant has amended the independent claims 1, 12 and 18 to include the 
limitation of wherein the three or more images are captured simultaneously, 

[Chen, VSMM99] does not explicitly disclose wherein the three images are 
captured simultaneously. 

Lu teaches the simultaneous capture of multiple images (column 3, lines 56-64 
and column 6, lines 1-10 and Fig. ) for use in an embodiment similar to that of [Chen 
VSMM99]. In the invention of Lu, the advantage of capturing the images at a single 
moment in time is advantageous in order to ensure that objects being captured do not 
move or shift between image capture from different images. Therefore it would have 
been obvious to one of ordinary skill in the art at the time of invention to capture multiple 
images simultaneously for generating 3D image data from matching the different 
images in order to ensure that the images represent the objects in the same position so 
that all of the images to be matched represent the same set of data defining the object 
in combination with the 3D depth recovery information disclosed by [Chen VSMM99]. 

The following is in regard to Claim 13. As shown above, the combination of 
[Chen, VSMM99] and Lu suggests a computer implementation of a volumetric stereo 
matching method that adequately satisfies the limitations of Claims 12. The disclosed 
volumetric stereo matching method operates on photographic images (e.g. [Chen, 
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VSMM99] Figs. 2 or 8). This necessitates some means of capturing images, such as a 
camera(s) (see [Chen, VSMM99] page 34, left column, paragraph 1 ). In order for the 
CPU to process the captured images, the camera would have to be somehow 
electrically coupled (e.g. via some camera interface) to the CPU. Therefore, the features 
of the image processing system proposed in Claim 13 are inherent to a computer-based 
implementation of the volumetric stereo matching method of [Chen, VSMM99]. 

The following is in regard to Claim 14. As shown above, the combination of 
[Chen, VSMM99] and Lu suggests a computer implementation of a volumetric stereo 
matching method that adequately satisfies the limitations of Claims 12. As mentioned 
above the "image matching module" and "propagation module" are resident on a 
memory. Modern memory is implemented as an integrated circuit. All modern CPUs are 
implemented as integrated circuits. In this manner, a computer-based implementation of 
the a volumetric stereo matching method of [Chen, VSMM99] would adequately satisfy 
the limitations of Claim 14. 

7776 following is in regard to Claims 15-17. Claim 15-17 recite essentially the 
same limitations as claim 5-7, respectively. Therefore, with regard to claim 15-17, 
remarks analogous to those presented above relative to claim 5-7 are respectively 
applicable. 
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The following is in regard to Claim 27. As shown above, [Chen, VSMM99] 
discloses a volumetric stereo matching method that adequately satisfies the limitations 
of Claims 24. Although [Chen, \/SMM99] does not explicitly mention associating the 
normalized cross-correlation (NCC) with a matching cost, the usage of the NCC in 
[Chen, \/SMM99] suggests that the two are closely related and could indeed be used 
interchangeably. As mentioned above, [Chen, VSMM99] uses the NCC, pa, v,cf within 
windows associated with each of the propagating fronts (e.g. windows centered about 
the voxel {u,v,d). on the surface fronts - [Chen, VSMM99] page 31 , left column, lines 1-3 
and right column, paragraph 2, sentence 1) to determine which of two coincident 
surface fronts to continue to propagate ([Chen, VSMM99] page 31 , right column, last 
paragraph, sentence 5). Furthermore, the NCC is indicative of the "correctness" of a 
match ([Chen, VSMM99], Section 3.2, paragraph 1 , sentence 1 ). All of this is suggests 
the that the NCC provides an adequate measure of the matching cost of the 
propagating surface fronts. Therefore, it would have been obvious to one of ordinary 
skill in the art, at the time of the applicant's claimed invention, to relate the matching 
cost of the propagating surface fronts to the computation of the NCC or designate the 
NCC as the matching cost. In either case, the motivation for doing so would have been 
to utilize the cross-correlative properties of the NCC to evaluate the degree to which 
voxels, comprising the surface fronts, represent correct matches. 

The following is in regard to Claim 28. As shown above, [Chen, VSMM99] 
discloses a volumetric stereo matching method that, when modified in the manner 
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suggested above, adequately satisfies the limitations of Claims 26. [Chen, VSMM99] 
suggests rectification of at least one pair of images corresponding to the base view of 
the scene and at least one of the other views (e.g. "reference views"). See [Chen, 
VSMM99] Section 2. Therefore, it would have been obvious to one of ordinary skill in 
the art, at the time of the applicant's claimed invention, to further perform rectification of 
at least one pair of images corresponding to the base view of the scene and at least one 
of the other views (e.g. "reference views"). The result of rectification is that "scanlines 
[epipolar lines] are parallel in each image, and the corresponding ones are collinear 
across the images" ([Chen, VSMM99] Section 2, paragraph 1, sentence 2 and last 
paragraph). This is known to have the effect of greatly simplifying the problem of finding 
correspondences across stereo images. 

Claims 2-3 and 19-20 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over the combination of [Chen, VSMM99] and U.S. Patent 5,852,672 to Lu, in view of 
[Zhang94] (Zhang, Deriche, Faugeras, and Luong, "A Robust Technique for Matching 
Two Uncalibrated Images Through the Recovery of Unknown Epipolar Geometry", 
INRIA1994). 

The following is in regard to Claim 2. As shown above, the combination of [Chen, 
VSMM99] and Lu discloses a volumetric stereo matching method that sufficiently 
conforms to the depth recovery method proposed in Claim 1 . The volumetric stereo 
matching method of [Chen, VSMM99] operates on multiple images (see, for instance 
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[Chen, VSMM99] Fig. 2 and Fig. 11). That is, the depth recovery according to this 

method includes the following: 

(2. a.) Detecting a plurality of image pixels in a first image corresponding 

to a first view of a scene. 
(2.b.) Detecting a plurality of image pixels in at least a second image 

corresponding to a respective at least a second view of the scene. 
The at least a second image deviates from the first image as a 
result of camera relative motion. This is typical of stereo matching 
algorithms (see also [Chen, VSMM99] page 34, Example 4). 

As noted above, the volumetric stereo matching method of [Chen, VSMM99], like most 

stereo matching methods, involves determining correspondences between a plurality of 

pixels in a first image and a plurality of pixels of at least a second image. In other words, 

the method of [Chen, VSMM99] further involves: 

(2.C.) Determining a first two-view correspondence between the plurality 
of detected image pixels in the first image and a plurality of 
detected image pixels in one of the at least a second image 
resulting in a first potential match set of candidate image pixels 
between the first image and the one of the at least a second image. 
This process involves the evaluation of the cross-correlation 
Du,v,das discussed above with regard to step (1.b.). 
[Chen, VSMM99] further suggests subsequently computing the projective 

reconstruction from the derived correspondence information in order to reconstruct the 
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projective 3D structure of the observed scene ([Chen, VSMM99] Section 4, paragraph 
1)- 

[Chen, \/SMM99] does not, however, expressly show or suggest: 

(2.d.) Concatenating a plurality of two-view match sets to form a first 
multiple-view potential match set; and 

(2.e.) Determining a second multiple-view potential match set that is a 
refinement of the first multiple-view potential match set. 

[Zhang94], on the other hand, discloses a stereo matching method. Stereo 
matching across disparate views of a scene is generally a fundamental step in depth 
recovery. The stereo matching method involves at least the following steps (refer to 
[Zhang94] Section 6,3 on pages 16-19): 

(2.C.) Determining an initial set of correspondences or matches (e.g. a 
first two-view correspondence) between the plurality of detected 
image pixels in the first image and a plurality of detected image 
pixels in one of the at least a second image resulting in a first 
potential match set of candidate image pixels (e.g. candidate 
matches - see, for example, [Zhang94] page 10, Section 5, 
paragraph 1 , lines 1-2) between the first image and the one of the 
at least a second image. 

(2.d.) Removing outliers from the initial set of matches (i.e. determining a 
"multiple-view correspondence" between the plurality of detected 
image pixels in the first image and the plurality of detected image 
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pixels in the at least a second image). See, for example, [Zhang94] 
page 18, line 24^, in conjunction with the equations for the weights 
Wi on pages 17-18. In this manner, the set of matches or 
correspondences thus obtained (e.g. the "multiple-view 
correspondences") represents a refinement of the "first two-view 
correspondence", resulting in a "second potential match set" of 
candidate image pixels between the first image and the at least a 
second image. 

(2.e.) The "second potential match set" is based at least in part on a 

computation of reprojection error (e.g. res/daa/ n - [Zhang94] page 
17, paragraphs 1-2) for matched pixels that resulted from "the 
difference between the /-th observation and its fitted value'"^'" 
([Zhang94] page 17, paragraph 1, lines 1-2). 
The teachings of [Chen, VSMM99] and [Zhang94] are combinable because they 
are analogous art. Specifically, both [Chen, VSMM99] and [Zhang94] disclose methods 
for stereo matching. Therefore, it would have been obvious to one of ordinary skill in the 
art, at the time of the applicant's claimed invention, to use the methodology of [Zhang94] 
" in particular, steps (2.d.)-(2.e.) above - to refine the initial set of correspondences 
derived according to the method of [Chen, \/SMM99]. It would have been obvious to 
one of ordinary skill in the art, at the time of the applicant's claimed invention, that, in 

^ Lines are counted sequentially from the top beginning with 1 . An equation is treated as a single line, regardless of its length. 

^ It should be understood that the fitted value in this case is a projection of the matched pixels into some view being evaluated and the 
observation is the respective candidate point correspondences {(lUij, niii)}. See [Zhang94] page 17, last paragraph. 
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applying this refinement process to method of [Chen, VSMM99], the fitted values 
become the reprojected projective points that delimit the projective structure of the 
observed scene (i.e. "matched pixels that resulted from a projective reconstruction of 
the first potential match set"). The motivation for using the method of [Zhang94], in this 
manner, would have been to detect and remove outliers (e.g. bad locations and/or false 
/77afc/?es - [Zhang94] page 16, last paragraph) from the initial set of candidate matches 
(e.g. those obtained via step (2.c.) above). See [Zhang94] Section 6.3. 

The following is in regard to Claims 19, Claim 19 recites essentially the same 
limitations as claim 2. (The 3D reconstruction instructions merely implement the depth 
recovery method of Claim 2). Therefore, with regard to claim 19, remarks analogous to 
those presented above relative to claim 2 are applicable. 

Ttie following is in regard to Claim 3. As shown above the methods of [Chen, 
VSMM99] and [Zhang94] can be combined to satisfy the limitations of Claim 2. 
[Zhang94] further disclose that the "outlier-less" set of correspondences (i.e. the 
"second potential match set") is based at least in part on a least median of squares 
computation (LMedS - [Zhang94] Abstract and page 17, lines 4-7 and paragraph 2) of 
the reprojection errors (i.e. residuals n- see above) related to matched pixels in the first 
potential match set. LMedS, as put forth by [Zhang94], has an advantage over other 
methods in that it "is very robust to false matches as well as outliers due to bad 
localization" ([Zhang94] page 17, paragraph 1, last sentence). Therefore, it would have 
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been obvious to one of ordinary skill in the art, at the time of the applicant's claimed 
invention, to use the LMedS of [Zhang94] to detect and remove outliers from the first 
potential match set. 

The following is in regard to Claim 20. Claim 20 recites essentially the same 
limitations as claim 3. (The 3D reconstruction instructions merely implement the depth 
recovery method of Claim 3). Therefore, with regard to claim 20, remarks analogous to 
those presented above relative to claim 3 are applicable. 

9. Claims 10-11 and are rejected under 35 U.S.C. 103(a) as being 
unpatentable over the combination of [Chen, VSMM99] and U.S. Patent 5,852,672 to 
Lu, in view of [Okutomi93] (Okutomi and Kanade "A Multiple-Baseline Stereo", IEEE 
1993), in further view of [Lewis95] (Lewis, "Fast Normalized Cross-Correlation", 1995). 

The following is in regard to Claim 10. As shown above, [Chen, \/SMM99] 
discloses a volumetric stereo matching method that sufficiently conforms to the depth 
recovery method proposed in Claim 9. As discussed above, the method of [Chen, 
VSMM99] involves the computation of the NCC between a first image window of a pre- 
determined size in the base view, and a second image window of the same pre- 
determined size in other views ([Chen, VSMM99] Section 3.1). It was also show above 
that the NCC is intimately related to the matching cost. Despite this, [Chen, VSMM99] 
does not expressly show or suggest that the matching cost is determined by computing 
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the summation of all the normalized cross-correlations between a first image window of 
a pre-determined size in the base view, and a second image window of the same pre- 
determined size in a reference view. 

[Okutomi93] disclose a stereo matching method that uses multiple stereo pairs 
with various baselines for the purposes of depth recovery ([Okutomi93] Abstract 
sentence 1). According to [Okutomi93], the summation of the sum of squared 
differences (SSD) from multiple stereo pairs can be used to indicate the "correctness" of 
a set of matching points (pixels). See [Okutomi93] page 353 (right column, paragraph 3, 
first and last sentence), page 354, (left column, lines 7-10), page 355 (Section B, 
paragraph 2, sentence 1), and equation (39) on page 362. This summation is analogous 
to a matching cost in that it approaches a minimum for correct matches ([Okutomi93] 
page 353, right column, paragraph 3, last sentence). According to equation (39) the 
computation is made within a predetermined window \N belonging to each of the stereo 
images ([Okutomi93] page 354, right column, paragraph 1, line 4). In summary, 
[Okutomi93] shows a "matching cost" determined by computing the summation of all the 
SSUs between a first image window of a pre-determined size in the base view and a 
second image window of the same pre-determined size in other views (e.g. "reference 
views"). Given this, it would have been obvious to one of ordinary skill in the art, at the 
time of the applicant's claimed invention, to use a "matching cost" (i.e. a measure of the 
similarity or "correctness" of a match) determined by computing the summation of all 
the SSD*s between a first image window of a pre-determined size in the base view and 
a second image window of the same pre-determined size in other views (e.g. "reference 
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views"). As pointed out by [Okutomi93], this would have the effect of advantageously 
reducing global mismatches ([Okutomi93], page 353, right column, paragraph 3, first 
sentence). 

Note that the SSD and NCC are closely related mathematically. As measures of 
similarity among regions of images, they are functionally equivalent. See [Lewis95] 
Section 2, paragraph 1 . Furthermore, it can also be shown that the process of 
minimizing the SSD and maximizing the NCC are in fact equivalent operations. All of 
this suggests that SSD and NCC are essentially interchangeable. Moreover, the NCC 
has the advantage over the SSD in that it is invariant to "changes in image amplitude 
such as those caused by changing lighting conditions across the image sequence" 
([Lewis95] page 2, lines 8-17). Since lighting conditions are likely to vary across the 
various stereo images, it would have been obvious to one of ordinary skill in the art, at 
the time of the applicant's claimed invention, to substitute the NCC for the SSD in the 
stereo matching method of [Okutomi93], This would yield a "matching cost" determined 
by computing the summation of all the NCC's between a first image window of a pre- 
determined size in the base view and a second image window of the same pre- 
determined size in other views (e.g. "reference views"). 

The teachings of [Chen, \/SMM99] and [Okutomi93] are combinable because 
they are analogous art, namely, the art of stereo matching. The teachings of [Lewis95] 
are applicable to both the methods of [Chen, VSMM99] and [Okutomi93] because all 
relate generally to correlative approaches for evaluating the similarity of windowed- 
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regions belonging to a collection of images of a scene. Combining these teachings as 
suggested above yields a method that conforms sufficiently to that of claim 10. 

The following is in regard to Claim 11. As shown above the teachings of [Chen, 
\/SMM99], [Okutomi93], and [Lewis95] can be combined so a to yield a method that 
adequately satisfies the limitations of Claim 10. [Chen, \/SMM99] suggests rectification 
of at least one pair of images corresponding to the base viev\^ of the scene and at least 
one of the other views (e.g. "reference views"). See [Chen, VSMM99] Section 2. 
Therefore, it would have been obvious to one of ordinary skill in the art, at the time of 
the applicant's claimed invention, to further perform rectification of at least one pair of 
images corresponding to the base view of the scene and at least one of the other views 
(e.g. "reference views"). The result of rectification is that "scanlines [epipolar lines] are 
parallel in each image, and the corresponding ones are collinear across the images" 
([Chen, \/SMM99] Section 2, paragraph 1 , sentence 2 and last paragraph). This is 
known to have the effect of greatly simplifying the problem of finding correspondences 
across stereo images. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Wes Tucker whose telephone number is 571-272-7427. 
The examiner can normally be reached on 9AM-5PM. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Bhavesh Mehta can be reached on 571-272-2214. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-21 7-91 97 (toll-free). 

Wes Tucker 
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